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[Title of Invention] INTEGRATED VIEW SYSTEM 

[Claims] 
[Claim 1] 

An integrated view system comprising: 

a stereo-camera installed in a vehicle for taking images of predetermined 
outside area; 

a stereo-image recognizer for processing a pair of images taken by the stereo-camera 
to recognize objects that are obstacles to the front, thus generating obstacle data; 

an integrated view data generator for generating integrated view data including 
three-dimensional view data based on the pair of images taken by the stereo-camera and 
the obstacle data from the stereo-image recognizer; and 

an integrated view displaying means for displaying the integrated view data as 
visible images to crews on the vehicle. 
[Claim 2] 

The integrated view system according to claim 1, the peripheral wide-ai'ea view 
data is added to the tluee-dimensional view data. 
[Claim 3] 

The integrated view system according to claim 1 or 2, wherein the integrated 
view displaying means is a head mount display for overlapping the visible images of the 
integrated view data and actual view. 
[Claim 4] 

The integrated view system according to claim 3, wherein the integrated view 
data generator is capable of removing the three-dimensional view data by the 
stereo-camera depending on the instruction inputs. 
[Claim 5] 

The integrated view system according to any one of claims 1 to 4 wherein the 
stereo-camera is composed of the two infrared cameras arranged as separated from each 
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other by a distance corresponding to a specific base line. 
[Claim 6] 

The integi*ated view system accordmg to any one of claims 1 to 4 further comprismg 
multiple stereo-cameras, which ai'e either an infrared camera, a milli-wave camera or an 
intensifier, different one another, and selectively used in accordance with actual views. 

[Detailed description of the invention] 
[0001] 

[Teclinical Field] 

The present invention relates to an integrated view system that provides crews 
on vehicle with views of high visibility even at actually low visibility. 
[0002] 

[Background Art] 

Vehicles, for example, aircraft ai*e provided with a view system having image 
sensors such as an injfrared camera, a milli-wave radar and a laser radar. The view system 
offers a driver or pilot with artificial pseudo-views generated based on view data collected 
by the image sensors at low visibility at night or in bad weather and three-dimensional 
(three-dimensional) map data stored in the system for safety. 
[0003] 

Japanese-Unexamined Patent Publication No. 72350/1999 discloses generation 
of pseudo-views using wide-ai'ea geographical data based on tliree-dimensional map 
stored in memoiy and data on obstacles such as' high-voltage electrical po\yer lines, 
skyscrapers and cranes and displaying pseudo-views and actual views overlapping with 
each other on a transparent-type display mounted on a helmet of a pilot. 
[0004] 

[Problems to be resolved by the invention] 

View data collected by the image sensors such as an infrared camera, a milli-wave 
radar and a laser radar are, however, not sufficient for a driver or pilot. Moreover, 
thi-ee-dimensional map data cannot follow actual changing geographical conditions. 
Pseudo-views generated based on these data therefore do not meet the requirements of a 
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driver or pilot. 
[0005] 

In detail, infrai-ed cameras can be used at ceitain level of low visibility, paiticulai'ly, 
can generate extremely clear images at night, however, lack in reality, perspective and 
feeling of speed due to monochrome images. Milli-wave radars can cover relatively 
long rage even in rainy weather, thus useful in image displaying at low visibility, 
however, cannot generate clear images due to wavelength extremely longer than light, 
thus not sufficient for a driver or pilot. Laser radars have an excellent obstacle detecting 
function, however, take long for scanning a wide area, thus revealing low response. 
For a narrow scanning ai*ea, they provide relatively clear images, but, narrow views for 
a driver or pilot, thus not sufficient for safety. 
[0006] 

Generation of images of scenery in wide range of sight which could be viewed 
by a driver or pilot is useful. Such image generation requires decision of degree of risk 
of collision based on comparison among geographical data, obstacle data and vehicle 
positional data (longitude, latitude and altitude). These data, however, may not match 
actual land features and obstacles. Such image generation thus has a difficulty in 
covering newly appearing obstacles and requires a lot of confirmation of safety. 
[0007] 

The present invention has been made considering the situations shown above. 
A pui'pose of the present invention is to provide an integrated view system tliat oflFers crews 
on a vehicle with almost real pseudo views at high visibility like in a good whether even 
at low visibility with detection of obstacles to tlie front for safe and sure flight or driving. 
[0008] 

[Means of solving the problems] 

For the purposes shown above, claim 1 of the present invention provides an 
integrated view system comprising: a stereo-camera installed in a vehicle for taking 
images of predetermined outside area; a stereo-image recognizer for processing a pair of 
images taken by tlie stereo-camera to recognize objects that are obstacles to the front, 
thus generating obstacle data; an integrated view data generator for generating integrated 
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view data including three-dimensional view data based on the pair of images taken by 
the stereo-camera and the obstacle data from the stereo-image recognizer; and an 
integrated view displaying means for displaying the integrated view data as visible 
images to crews on the vehicle. 
[0009] 

In claim 2, the integrated view data generator of claim 1 adds the peripheral 
wide-area view data to the tluee-dimensional view data by the stereo-camera. 
[0010] 

In claim 3, the integrated view displaying means in claim 1 or 2 is the 
transparent-type head mount display for overlapping the visible images of the integrated 
view data of the integrated view data generator and actual view. 
[0011] 

In claim 4, the integrated view data generator of claim 3 is capable of removing 
the three-dimensional view data from the stereo-camera depending on the instruction 
inputs. 
[0012] 

In claim 5, the stereo-camera in any one of claims 1 to 4 is the two infrared 
cameras arranged as separated from each otlier by a distance corresponding to a specific 
base line. 
[0013] 

In claim 6, the multiple stereo cameras in any one of claims 1 to 4 are installed, 
either an infrared camera, a milli-wave camera or an intensifier, different from one 
another and selectively used in accordance with actual views. 
[0014] 

In claim 1 of the present invention, a stereo-camera is installed in a vehicle, the 
integrated view data ai-e generated by the three-dimensional view data processed by a pair 
of tlie images taken by the stereo-camera and the obstacle data processed by a pair of images 
taken by the same stereo-camera, and displayed as visible images to crews on the vehicle. 
[0015] 

In claim 2, the peripheral wide-area view data is added to the three-dimensional 
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view data view data by the stereo-camera. In claim 3, the transparent-type head mount 
display that can overlap the integrated view data and actual views is used. In claim 4, it 
is possible to remove the tliree-dimensional view data from the stereo-camera, and 
selectively overlap the integrated views including obstacle data obtained by removing 
the tliree-dimensional view data from the actual views, and the integrated views 
including the three dimensional data and obstacle data. In claim 5, the stereo-camera 
is configured by the two infrared cameras arranged as sepai'ated from each other by a 
distance corresponding to a specific base line. And in claim 6, the multiple stereo 
cameras, either an infrared camera, a milli-wave camera or an intensifier, different from 
one another and selectively used in accordance with actual views. 
[0016] 

[Detailed Description of Preferred Embodiments] 

Preferred embodiments according to the present invention will be disclosed 
with reference to the attached drawings. Fig. 1 and 2 relate to one embodiment of the 
present invention. Fig. 1 shows a block diagram of an integrated view system according 
to the present invention. Fig. 2 illustrates displaying zones. 
[0017] 

An integrated view system 1 shown in Fig. 1 is installed in a vehicle such as an 
automobile, a train, and an aircraft. The system 1 offers integrated views to a driver or 
pilot generated as visible images of virtual reality at high visibility like in good weather 
even tliough actual visibility is very low in bad weather due to mist or fog, or at night. 
Disclosed hereinafter is an embodiment in which the integrated view system 1 is mstalled in 
an aircraft such as a helicopter that flies at relatively low altitude. 
[0018] 

Hie integrated view system 1 is provided v^th a stereo-camera 2 for taking images 
of forward scenery of a predetermined area, an image combining apparatus 10 and an 
integrated view displaying apparatus 20 as main components. 

A pair of left and right images taken by the stereo-camera 2 is displayed on left 
and right viewing points of a pilot to generate three-dimensional (three-dimensional) 
images giving perspective and feeling of altitude and speed to the pilot. Moreover, the 
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pair of left and right images is processed by stereo-image processing for calculation of 
data on (relative) distance to objects. The image and distance data are processed by 
image recognition processing for displaying obstacles as warning when the obstacles are 
detected on or in the vicinity of flight route. The integrated view system 1 is provided 
with a sight-axis switch 3 for varying an axis of sighting by which the stereo-camera 2 
tums into the direction required by the pilot or other crews, a display-mode switch 4 for 
controlling the stereo camera 2 to halt displaying tliree-dimensional images, and flight 
data interface 5 for entering flight data such as speed, altitude, position and attitude of a 
helicopter. 
[0019] 

The sight-axis switch 3 is useful to know beforehand the conditions of flight 
route to which the helicopter is to turn into or determine whether there is any obstacle 
on the route. The switch 3 in tliis embodiment is a manual switch to manually rotate tlie 
optical axis of the stereo-camera 2. Not only tliis, the stereo-camera 2 can be automatically 
turned into any direction by detecting pilot's viewing point by a head-motion tracker 23, 
etc., which will be described later. 
[0020] 

The stereo-camera 2 in this embodiment consists of two infrared cameras for 
generating extremely clear images particularly at night. The two infrared cameras are 
arranged with an optimum distance (base-line length) within an allowable range based 
on search range and distance accuracy for accurately detecting obstacles predicted under 
several flight conditions. Flight conditions under which a pilot requires support of 
artificial view at general low visibility mostly include night flight or other flight very 
close to this. Infrared cameras having an excellent night view function are useful in such 
conditions. Particularly, three-dimensional images generated by two infrared cameras 
offer virtual reality to a pilot with perspective and feeling of altitude and speed which 
cannot be achieved by a single infrai'ed camera. Other flight conditions can be covered 
by several types of image sensors such as an ordinary camera, an intensifier that 
responses faint light, active/passive milli-wave cameras exliibiting excellent transparency to 
mist and rain and sensitive CCDs, other than infrared cameras. These sensors can be 
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selectively combined in accordance with cause of low visibility. It is also possible to 

select multiple types of cameras or image sensors depending on conditions. 

[0021] 

When relatively light-weight image sensors are used for cameras of the stereo 
camera 2, the base-line lengtlr for the stereo-camera 2 can be set by shifting both or 
either of cameras with the image sensors. Heavy image sensors can be fixed with an 
objective lens mechanism installed in a tube of a periscope, the tube-length being variable 
for varying the base-line length, 
[0022] 

In paiticular, the image combining appai*atus 10 is provided with a stereo-image 
recognition processor 1 1 for recognizing obstacles by processing left and right images from 
the stereo cainera 2, a geographical image generator 12 for generating three-dimensional 
geographical images of scenery which could be viewed by a pilot or crews based on 
view-point data sent from the head-motion tracker 23 which will be described later and 
data from the flight data interface 5, and an integrated view generator 13 for generating 
integrated views which are combination of tluee-dimensional view data of left and right 
images from the stereo-camera 2, obstacle data from the stereo-image recognition 
processor 11, geographical data on wide view from the geographical image generator 12 
and data from the flight data interface based on the statuses instructed by tlie sight-axis 
switch 3 and display-mode switch 4 of the stereo camera 2. 
[0023] 

The stereo-image recognition processor 1 1 is provided with an image database 
11a that stores several types of three-dimensional data for recognizing and displaying 
several types of obstacles, etc. The geographical image generator 12 is provided with 
a three-dimensional digital map 12a that stores wide-area geographical data obtained by 
aerial survey or from satetHtes. 
[0024] 

The integrated view displaying apparatus 20 is provided with a head-mount 
display (HMD) 21 to be mounted on a helmet of the pilot or crews and having a 
transpai-ent-type display such as a transparent-type liquid-crystal display panel by which 
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a pilot or crews caii view actual scenery tluough integrated views from tlie integrated 
view generator 13, a display adjuster 22 for adjusting intensity and contrast of 
integrated views, and transparency to the actual views on the HMD 21 so that the pilot 
or crews can observe the overlapped actual views and integrated views in a good 
condition, and the head-motion tracker 23 for tracking the head position and attitude of 
the pilot or crews to output view-point data of the pilot or crews. 
[0025] 

In the integrated view system 1 in the configuration as above, a pair of left and 
right images taken by the stereo-camera 2 during flight is sent to the integrated view 
generator 13 as three-dimensional images and also to the stereo-image recognition 
processor 11 for detecting forwai'd obstacles. The stereo-image recognition processor 
11 processes the left and right images from the stereo-camera 2 by stereo -matching 
processing to obtain correlation between the images, thus calculating distance data by 
triangular surveying based on parallax to the same object, the position of the stereo 
camera 2 and its parameter such as focal length. 
[0026] 

Data stored in the image database is accessed based on the distance and image 
data for recognizing obstacles, any objects that would block flight. The integrated 
view system 1 installed in aircraft, for example, helicopters that fly relatively low 
altitude recognizes structures such as pylons and skyscrapers or other aircraft, etc., in 
forwai'd view during night flight. When the system 1 also recognizes high-voltage 
electrical power lines in the image data under recognition of pylons, it generates 
obstacle data that symbolizes or emphasizes the power lines and sends the data to the 
integrated view generator 13. 
[0027] 

The present invention therefore offers high safety flight or driving without 
determination of degree of risk of collision by comparing stored positional data such as 
longitude, latitude and altitude with actual flight data in tlxree-dimensional digital map 
as the conventional arts. Accordingly, when the integrated view system recognizes 
pylons, high-voltage electrical power lines and skyscraper, etc., in forward view via the 
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stereo camera 2, it immediately warns the pilot of those structures as obstacles that 
could collide with so that the helicopter immediately takes an evasive action. Thus, 
the safety can be enlianced veiy effectively. 
[0028] 

The geographical image generator 12 performs coordinate-conversion of 
aircraft data such as speed, altitude and attitude and flight positional data input via the 
aircraft flight data interface 5 onto viewing points of the pilot or crews input from the 
head-motion tracker 23. The generator 12 fuilher retrieves the converted data from the 
thi'ee-dimensional digital map 12a as tliree-dimensional geographical images which 
could be viewed by the pilot or crews and sent the three-dimensional images to the 
integrated view generator 13. The tlii-ee-dimensional images are geographical data 
wider than actual scenery in forwai-d view, for example, a row of mountains in the 
distance which will not be directly comiected to safety against collision. These 
three-dimensional geographical image data can been seen as almost real scenes based on 
tliree-dimensional display generated by CG (computer graphics) with geographical data 
such as place names, lakes, loads and rivers if necessaiy. 
[0029] 

The tluee-dimensional geographical image data may be generated from images 
detected by a wide milli-wave radar instead of the tliree-dimensional digital map 12a. 
[0030] 

The integrated view generator 1 3 receives tliree-dimensional image data of left 
and right images taken by the stereo-camera 2 controlled by the sight-axis switch 3 and 
also the obstacle images generated by the stereo-image recognition processor 11. The 
generator 13 also receives wide peripheral geographical images generated by tlie geograpliical 
image generator 12. The integrated view generator 13 combines the obstacle and peripheral 
geographical images with image processing such as adjustments to resolution, intensity, 
contrast and color, and also edge-blending under the control of the display-mode switch 
4, to generate natural images with no visible joints of combination as integrated view 
data. The integrated view generator 1 3 combines the integrated view data with flight 
data such as speed, altitude, position and attitude sent from the flight data interface 5 if 



9 



necessary, and sends them to the HMD 21. At high visibility such as a good weather, 
the stereo camera 2 may be turned off so tliat integrated view data not three-dimensional 
images of the stereo camera 2 but the obstacle data and other data, if necessary, are sent 
to the HMD 21. 
[0031] 

Fig. 2 illustrates viewing zones covered by the HMD 21 with the integrated 
view data processed as disclosed above. A viewing zone surrounded by a dashed line 
is used for displaying the forward view data with obstacle data from the stereo camera 2. 
Another viewing zone surrounded by a dotted line but outside the dashed line is used for 
displaying the wide-area view data of three-dimensional geographical images. The 
pilot or crews can see actual scenery from a cockpit 30 through windows overlapping 
with the integrated views while watching indicators on the cockpit, thus the pilot or 
crews can see to indicated data in addition to those displayed on the HMD 21 . 
[0032] 

As disclosed above, the integrated view system 1 according to the present 
invention offers a pilot or crews witli tliree-dimensional images almost real in perspective, 
altitude and speed based on left and right images taken by the stereo camera 2 even at a 
low visibility, which camiot be achieved by a single camera. The integrated view 
system 1 further processes tlie left and right images from the stereo camera 2 by stereo-image 
processing for image recognition using distance data to detect obstacles on or in the 
vicinity of flight rout and displays the obstacles as warning. 
[0033] 

The integrated view system 1 according to the present invention thus offers 
pseudo-visual flight even at row visibility to support a pilot for safe and sure flight in 
regular service or emergency. Thi*ee-D images may not be required at high visibility, 
however, detection of obstacles for warning by stereo-image recognition processing 
achieves further safe flight. Tlii*ee-dimensional image display of forward views and 
obstacle detection/warning are performed by the stereo-camera 2 with no sensors for 
respective functions. The integrated view system 1 according to present invention thus 
can be structured as a reliable and light system at a low cost. 



10 



[0034] 

[Effect of the Invention] 

As disclosed above, the present invention offers crews on a vehicle with almost 
real pseudo views at high visibility like in a good whetlier even at low visibility for safe 
and sure flight or driving with detection of obstacles to the front. 

[Brief Description of the Drawings] 
[Fig. 1] 

Fig. 1 shows a block diagram of an integrated view system according to the 
present invention; and 
[Fig. 2] 

Fig. 2 illustrates displaying zones of the integrated view data. 

[Description of the Reference Numerals] 

1 . Integrated view system 

2. Stereo cainera 

1 0. Image combining apparatus 

1 1 . Stereo-image recognition processor 
1 3 . Integrated view generator 

20. Integrated view displaying apparatus 

21. Head-mount display 
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[Document Type] ABSTRACT 

[Abstract] 

[Object] 

A purpose of the present invention is to provide an integrated view system that 
offers crews on a vehicle with ahnost real pseudo views at high visibility like in a good 
whether even at low visibility with detection of obstacles to the front for safe and sure 
flight or driving. 

[Means for Solving the Problems] 

A pair of left and right images taken by the stereo-camera 2 is sent to the 
integrated view generator 13 as tliree-dimensional images and also to the stereo-image 
recognition processor 11 for detecting forward obstacles. At the same tune, the geographical 
image generator 12 generates wide peripheral geographical images as tliree-dimensional 
geographical data. The integrated view generator 13 generates natui-al integrated view 
data by the thi'ee-dimensional images and obstacle data, and peripheral geographical images. 
Further, the integrated view generator 13 combines the integrated view data with flight 
data such as speed, altitude, position and attitude sent from the flight data interface 5 
and sends them to the HMD 21 . 

[Selected drawing] 
Fig. 1 



